dreamfield-3D¶

A toolkit to generate 3D mesh model / video / nerf instance / multiveiw images of colourful 3D objects by text and image prompts input. Edited by Shengyu Meng (Simon)
Wellcome to share the outputs in social media with tag #dreamfields3D 😀

Update logs¶

Beta v0.65 ：add options for apply image prompts only in assigned direction to avoid overfitting; could skip saving depth maps in training and testing; output sequence images in validation process. (2022-10-03)
Beta v0.60 : apply random fovy (view angle) in training; update image augmentation before feed into CLIP; improve training stability and performance. (2022-09-25)

Credits & Changelog¶

Dreamfields-3D is modified from ashawkey's dreamfields-torch, and its forks of IV Pravdin and Pollinations.AI, released under MIT License.

The main improvements in this notebook & repository were done by Shengyu Meng (Simon), including allow visualizing training process in colab, export 360° video and 3D mesh model with vertex colour, and more arguments.This code is released under MIT License

The original dreamfields was issued under Apache-2.0 license, proposed by Jain, Ajay and Mildenhall, Ben and Barron, Jonathan T. and Abbeel, Pieter and Poole, Ben in their paper, Zero-Shot Text-Guided Object Generation with Dream Fields published on CVPR 2022. The main different of dreamfields-torch compared with original dreamfields is, dreamfields-torch applied the torch version of instant-ngp to replace the original NeRF as backend, and re-write most of the codes.

The technical bases of original dreamfields are NeRF: Neural Radiance Fields, released under MIT License, proposed by Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng in their paper NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis published on ECCV 2020, and CLIP: Connecting Text and Images model developed by OpenAI, released under MIT License.

check the machine¶

In [1]:
#@title ### Check GPU
!nvidia-smi

Tue Mar 28 04:08:42 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.12    Driver Version: 525.85.12    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA A100-SXM...  Off  | 00000000:00:04.0 Off |                    0 |
| N/A   29C    P0    44W / 400W |      0MiB / 40960MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
In [2]:
#@title ### Mount Google drive (otpional)
from google.colab import drive
drive.mount('/content/drive')

Mounted at /content/drive

setup¶

In [3]:
#@title ### pull the repostory & get working dir
import os 
!git clone https://github.com/shengyu-meng/dreamfields-3D
%cd dreamfields-3D
DC_dir = os.getcwd()

Cloning into 'dreamfields-3D'...
remote: Enumerating objects: 313, done.
remote: Counting objects: 100% (158/158), done.
remote: Compressing objects: 100% (80/80), done.
remote: Total 313 (delta 103), reused 124 (delta 76), pack-reused 155
Receiving objects: 100% (313/313), 47.12 MiB | 23.31 MiB/s, done.
Resolving deltas: 100% (165/165), done.
/content/dreamfields-3D
In [4]:
#@title install dependencies
!pip install -r requirements.txt
!bash scripts/install_ext.sh
!bash scripts/install_PyMarchingCubes.sh

Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Collecting git+https://github.com/openai/CLIP.git (from -r requirements.txt (line 24))
  Cloning https://github.com/openai/CLIP.git to /tmp/pip-req-build-0loqucp2
  Running command git clone --filter=blob:none --quiet https://github.com/openai/CLIP.git /tmp/pip-req-build-0loqucp2
  Resolved https://github.com/openai/CLIP.git to commit a9b1bf5920416aaeaec965c25dd9e8f98c864f16
  Preparing metadata (setup.py) ... done
Collecting torch-ema
  Downloading torch_ema-0.3-py3-none-any.whl (5.5 kB)
Collecting ninja
  Downloading ninja-1.11.1-py2.py3-none-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (145 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 146.0/146.0 KB 8.4 MB/s eta 0:00:00
Collecting trimesh
  Downloading trimesh-3.21.2-py3-none-any.whl (680 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 680.5/680.5 KB 42.2 MB/s eta 0:00:00
Requirement already satisfied: opencv-python in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 4)) (4.7.0.72)
Collecting tensorboardX
  Downloading tensorboardX-2.6-py2.py3-none-any.whl (114 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 114.5/114.5 KB 16.7 MB/s eta 0:00:00
Requirement already satisfied: torch in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 6)) (1.13.1+cu116)
Requirement already satisfied: numpy in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 7)) (1.22.4)
Requirement already satisfied: pandas in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 8)) (1.4.4)
Requirement already satisfied: tqdm in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 9)) (4.65.0)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 10)) (3.7.1)
Requirement already satisfied: rich in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 13)) (13.3.2)
Collecting dearpygui
  Downloading dearpygui-1.9.0-cp39-cp39-manylinux1_x86_64.whl (2.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.8/2.8 MB 84.6 MB/s eta 0:00:00
Requirement already satisfied: packaging in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 16)) (23.0)
Requirement already satisfied: scipy in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 17)) (1.10.1)
Collecting lpips
  Downloading lpips-0.1.4-py3-none-any.whl (53 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.8/53.8 KB 7.5 MB/s eta 0:00:00
Requirement already satisfied: imageio in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 19)) (2.25.1)
Requirement already satisfied: imageio-ffmpeg in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 20)) (0.4.8)
Collecting pymeshlab
  Downloading pymeshlab-2022.2.post3-cp39-cp39-manylinux1_x86_64.whl (59.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 59.3/59.3 MB 21.0 MB/s eta 0:00:00
Collecting ipyplot
  Downloading ipyplot-1.1.1-py3-none-any.whl (13 kB)
Requirement already satisfied: IPython in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 23)) (7.9.0)
Requirement already satisfied: protobuf<4,>=3.8.0 in /usr/local/lib/python3.9/dist-packages (from tensorboardX->-r requirements.txt (line 5)) (3.19.6)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.9/dist-packages (from torch->-r requirements.txt (line 6)) (4.5.0)
Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.9/dist-packages (from pandas->-r requirements.txt (line 8)) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.9/dist-packages (from pandas->-r requirements.txt (line 8)) (2022.7.1)
Requirement already satisfied: importlib-resources>=3.2.0 in /usr/local/lib/python3.9/dist-packages (from matplotlib->-r requirements.txt (line 10)) (5.12.0)
Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib->-r requirements.txt (line 10)) (3.0.9)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.9/dist-packages (from matplotlib->-r requirements.txt (line 10)) (0.11.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib->-r requirements.txt (line 10)) (1.4.4)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib->-r requirements.txt (line 10)) (1.0.7)
Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.9/dist-packages (from matplotlib->-r requirements.txt (line 10)) (8.4.0)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.9/dist-packages (from matplotlib->-r requirements.txt (line 10)) (4.39.2)
Requirement already satisfied: markdown-it-py<3.0.0,>=2.2.0 in /usr/local/lib/python3.9/dist-packages (from rich->-r requirements.txt (line 13)) (2.2.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.9/dist-packages (from rich->-r requirements.txt (line 13)) (2.14.0)
Requirement already satisfied: torchvision>=0.2.1 in /usr/local/lib/python3.9/dist-packages (from lpips->-r requirements.txt (line 18)) (0.14.1+cu116)
Collecting shortuuid
  Downloading shortuuid-1.0.11-py3-none-any.whl (10 kB)
Requirement already satisfied: pickleshare in /usr/local/lib/python3.9/dist-packages (from IPython->-r requirements.txt (line 23)) (0.7.5)
Requirement already satisfied: traitlets>=4.2 in /usr/local/lib/python3.9/dist-packages (from IPython->-r requirements.txt (line 23)) (5.7.1)
Requirement already satisfied: prompt-toolkit<2.1.0,>=2.0.0 in /usr/local/lib/python3.9/dist-packages (from IPython->-r requirements.txt (line 23)) (2.0.10)
Collecting jedi>=0.10
  Downloading jedi-0.18.2-py2.py3-none-any.whl (1.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 80.1 MB/s eta 0:00:00
Requirement already satisfied: decorator in /usr/local/lib/python3.9/dist-packages (from IPython->-r requirements.txt (line 23)) (4.4.2)
Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.9/dist-packages (from IPython->-r requirements.txt (line 23)) (67.6.0)
Requirement already satisfied: backcall in /usr/local/lib/python3.9/dist-packages (from IPython->-r requirements.txt (line 23)) (0.2.0)
Requirement already satisfied: pexpect in /usr/local/lib/python3.9/dist-packages (from IPython->-r requirements.txt (line 23)) (4.8.0)
Collecting ftfy
  Downloading ftfy-6.1.1-py3-none-any.whl (53 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.1/53.1 KB 7.6 MB/s eta 0:00:00
Requirement already satisfied: regex in /usr/local/lib/python3.9/dist-packages (from clip==1.0->-r requirements.txt (line 24)) (2022.10.31)
Requirement already satisfied: zipp>=3.1.0 in /usr/local/lib/python3.9/dist-packages (from importlib-resources>=3.2.0->matplotlib->-r requirements.txt (line 10)) (3.15.0)
Requirement already satisfied: parso<0.9.0,>=0.8.0 in /usr/local/lib/python3.9/dist-packages (from jedi>=0.10->IPython->-r requirements.txt (line 23)) (0.8.3)
Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.9/dist-packages (from markdown-it-py<3.0.0,>=2.2.0->rich->-r requirements.txt (line 13)) (0.1.2)
Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.9/dist-packages (from prompt-toolkit<2.1.0,>=2.0.0->IPython->-r requirements.txt (line 23)) (1.16.0)
Requirement already satisfied: wcwidth in /usr/local/lib/python3.9/dist-packages (from prompt-toolkit<2.1.0,>=2.0.0->IPython->-r requirements.txt (line 23)) (0.2.6)
Requirement already satisfied: requests in /usr/local/lib/python3.9/dist-packages (from torchvision>=0.2.1->lpips->-r requirements.txt (line 18)) (2.27.1)
Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.9/dist-packages (from pexpect->IPython->-r requirements.txt (line 23)) (0.7.0)
Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.9/dist-packages (from requests->torchvision>=0.2.1->lpips->-r requirements.txt (line 18)) (2.0.12)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/dist-packages (from requests->torchvision>=0.2.1->lpips->-r requirements.txt (line 18)) (2022.12.7)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/dist-packages (from requests->torchvision>=0.2.1->lpips->-r requirements.txt (line 18)) (3.4)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.9/dist-packages (from requests->torchvision>=0.2.1->lpips->-r requirements.txt (line 18)) (1.26.15)
Building wheels for collected packages: clip
  Building wheel for clip (setup.py) ... done
  Created wheel for clip: filename=clip-1.0-py3-none-any.whl size=1369398 sha256=482fb35cd3e903cdc3e3686d2ba151d2f2f0649a69eb1bea35574de71feed9b3
  Stored in directory: /tmp/pip-ephem-wheel-cache-kz_pfq1b/wheels/c8/e4/e1/11374c111387672fc2068dfbe0d4b424cb9cdd1b2e184a71b5
Successfully built clip
Installing collected packages: ninja, trimesh, tensorboardX, shortuuid, pymeshlab, jedi, ftfy, dearpygui, torch-ema, lpips, ipyplot, clip
Successfully installed clip-1.0 dearpygui-1.9.0 ftfy-6.1.1 ipyplot-1.1.1 jedi-0.18.2 lpips-0.1.4 ninja-1.11.1 pymeshlab-2022.2.post3 shortuuid-1.0.11 tensorboardX-2.6 torch-ema-0.3 trimesh-3.21.2
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Processing /content/dreamfields-3D/raymarching
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: raymarching
  Building wheel for raymarching (setup.py) ... done
  Created wheel for raymarching: filename=raymarching-0.0.0-cp39-cp39-linux_x86_64.whl size=2812222 sha256=e96f61911824536f03f6a62ed18d36a9200ca445a8ef435c0efad2b2ae92cb83
  Stored in directory: /tmp/pip-ephem-wheel-cache-_5cnakc6/wheels/39/89/49/3e28339c4b4f791a9aacb61bddaa8c46fda0958da7bb002461
Successfully built raymarching
Installing collected packages: raymarching
Successfully installed raymarching-0.0.0
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Processing /content/dreamfields-3D/shencoder
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: shencoder
  Building wheel for shencoder (setup.py) ... done
  Created wheel for shencoder: filename=shencoder-0.0.0-cp39-cp39-linux_x86_64.whl size=2628004 sha256=1edd95f5178afd65aa4b8e25bd9071cc3dbf9a531f89b400f1e2197b99cfa2fb
  Stored in directory: /tmp/pip-ephem-wheel-cache-mmw2u1d1/wheels/de/27/49/a444fb2e4ffd652437a4f6b66f7af43d054ea49299e17fff1e
Successfully built shencoder
Installing collected packages: shencoder
Successfully installed shencoder-0.0.0
Cloning into 'PyMarchingCubes'...
remote: Enumerating objects: 331, done.
remote: Total 331 (delta 0), reused 0 (delta 0), pack-reused 331
Receiving objects: 100% (331/331), 609.82 KiB | 1.36 MiB/s, done.
Resolving deltas: 100% (184/184), done.
Cloning into 'eigen'...
remote: Enumerating objects: 118903, done.
remote: Counting objects: 100% (582/582), done.
remote: Compressing objects: 100% (209/209), done.
remote: Total 118903 (delta 395), reused 542 (delta 373), pack-reused 118321
Receiving objects: 100% (118903/118903), 103.17 MiB | 35.21 MiB/s, done.
Resolving deltas: 100% (98230/98230), done.
Note: switching to 'tags/3.4.0'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 3147391d9 Change version to 3.4.0.
/usr/local/lib/python3.9/dist-packages/setuptools/__init__.py:85: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated. Requirements should be satisfied by a PEP 517 installer. If you are using pip, you can try `pip install --use-pep517`.
  dist.fetch_build_eggs(dist.setup_requires)
running install
/usr/local/lib/python3.9/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
Compiling marching_cubes/src/_mcubes.pyx because it changed.
[1/1] Cythonizing marching_cubes/src/_mcubes.pyx
/usr/local/lib/python3.9/dist-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /content/dreamfields-3D/PyMarchingCubes/marching_cubes/src/_mcubes.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)
/usr/local/lib/python3.9/dist-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
running bdist_egg
running egg_info
creating PyMarchingCubes.egg-info
writing PyMarchingCubes.egg-info/PKG-INFO
writing dependency_links to PyMarchingCubes.egg-info/dependency_links.txt
writing top-level names to PyMarchingCubes.egg-info/top_level.txt
writing manifest file 'PyMarchingCubes.egg-info/SOURCES.txt'
reading manifest file 'PyMarchingCubes.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'images/smoothing.png'
warning: no files found matching 'README.rst'
adding license file 'LICENSE'
writing manifest file 'PyMarchingCubes.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib.linux-x86_64-3.9
creating build/lib.linux-x86_64-3.9/marching_cubes
copying marching_cubes/smoothing.py -> build/lib.linux-x86_64-3.9/marching_cubes
copying marching_cubes/exporter.py -> build/lib.linux-x86_64-3.9/marching_cubes
copying marching_cubes/__init__.py -> build/lib.linux-x86_64-3.9/marching_cubes
copying marching_cubes/numpy_smoothing.py -> build/lib.linux-x86_64-3.9/marching_cubes
running build_ext
building 'marching_cubes._mcubes' extension
creating build/temp.linux-x86_64-3.9
creating build/temp.linux-x86_64-3.9/marching_cubes
creating build/temp.linux-x86_64-3.9/marching_cubes/src
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Imarching_cubes/src -I/usr/local/lib/python3.9/dist-packages/numpy/core/include -I/usr/include/python3.9 -c marching_cubes/src/_mcubes.cpp -o build/temp.linux-x86_64-3.9/marching_cubes/src/_mcubes.o -std=c++11 -Wall
In file included from /usr/local/lib/python3.9/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1960,
                 from /usr/local/lib/python3.9/dist-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /usr/local/lib/python3.9/dist-packages/numpy/core/include/numpy/arrayobject.h:5,
                 from marching_cubes/src/_mcubes.cpp:795:
/usr/local/lib/python3.9/dist-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   17 | #warning "Using deprecated NumPy API, disable it with " \
      |  ^~~~~~~
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Imarching_cubes/src -I/usr/local/lib/python3.9/dist-packages/numpy/core/include -I/usr/include/python3.9 -c marching_cubes/src/marchingcubes.cpp -o build/temp.linux-x86_64-3.9/marching_cubes/src/marchingcubes.o -std=c++11 -Wall
In file included from marching_cubes/src/marchingcubes.cpp:2:
marching_cubes/src/marchingcubes.h: In function ‘void mc::marching_cubes(const vector3&, const vector3&, int, int, int, formula, double, std::vector<double>&, std::vector<typename vector3::size_type>&)’:
marching_cubes/src/marchingcubes.h:56:11: warning: typedef ‘using coord_type = typename vector3::value_type’ locally defined but not used [-Wunused-local-typedefs]
   56 |     using coord_type = typename vector3::value_type;
      |           ^~~~~~~~~~
marching_cubes/src/marchingcubes.h:57:11: warning: typedef ‘using size_type = typename vector3::size_type’ locally defined but not used [-Wunused-local-typedefs]
   57 |     using size_type = typename vector3::size_type;
      |           ^~~~~~~~~
In file included from marching_cubes/src/marchingcubes.cpp:2:
marching_cubes/src/marchingcubes.h: In function ‘void mc::marching_cubes_color(const vector3&, const vector3&, int, int, int, formula_sdf, formula_color, double, std::vector<double>&, std::vector<typename vector3::size_type>&)’:
marching_cubes/src/marchingcubes.h:519:11: warning: typedef ‘using coord_type = typename vector3::value_type’ locally defined but not used [-Wunused-local-typedefs]
  519 |     using coord_type = typename vector3::value_type;
      |           ^~~~~~~~~~
marching_cubes/src/marchingcubes.h:520:11: warning: typedef ‘using size_type = typename vector3::size_type’ locally defined but not used [-Wunused-local-typedefs]
  520 |     using size_type = typename vector3::size_type;
      |           ^~~~~~~~~
marching_cubes/src/marchingcubes.h: In function ‘void mc::marching_cubes_super_sampling(const vector3&, const vector3&, int, int, int, int, int, int, formula, formulaX, formulaY, formulaZ, double, std::vector<double>&, std::vector<typename vector3::size_type>&)’:
marching_cubes/src/marchingcubes.h:800:11: warning: typedef ‘using coord_type = typename vector3::value_type’ locally defined but not used [-Wunused-local-typedefs]
  800 |     using coord_type = typename vector3::value_type;
      |           ^~~~~~~~~~
marching_cubes/src/marchingcubes.h:801:11: warning: typedef ‘using size_type = typename vector3::size_type’ locally defined but not used [-Wunused-local-typedefs]
  801 |     using size_type = typename vector3::size_type;
      |           ^~~~~~~~~
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Imarching_cubes/src -I/usr/local/lib/python3.9/dist-packages/numpy/core/include -I/usr/include/python3.9 -c marching_cubes/src/pywrapper.cpp -o build/temp.linux-x86_64-3.9/marching_cubes/src/pywrapper.o -std=c++11 -Wall
In file included from /usr/local/lib/python3.9/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1960,
                 from /usr/local/lib/python3.9/dist-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /usr/local/lib/python3.9/dist-packages/numpy/core/include/numpy/arrayobject.h:5,
                 from marching_cubes/src/pyarraymodule.h:11,
                 from marching_cubes/src/pywrapper.h:6,
                 from marching_cubes/src/pywrapper.cpp:2:
/usr/local/lib/python3.9/dist-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   17 | #warning "Using deprecated NumPy API, disable it with " \
      |  ^~~~~~~
In file included from marching_cubes/src/pywrapper.cpp:4:
marching_cubes/src/marchingcubes.h: In function ‘void mc::marching_cubes(const vector3&, const vector3&, int, int, int, formula, double, std::vector<double>&, std::vector<typename vector3::size_type>&)’:
marching_cubes/src/marchingcubes.h:56:11: warning: typedef ‘using coord_type = typename vector3::value_type’ locally defined but not used [-Wunused-local-typedefs]
   56 |     using coord_type = typename vector3::value_type;
      |           ^~~~~~~~~~
marching_cubes/src/marchingcubes.h:57:11: warning: typedef ‘using size_type = typename vector3::size_type’ locally defined but not used [-Wunused-local-typedefs]
   57 |     using size_type = typename vector3::size_type;
      |           ^~~~~~~~~
In file included from marching_cubes/src/pywrapper.cpp:4:
marching_cubes/src/marchingcubes.h: In function ‘void mc::marching_cubes_color(const vector3&, const vector3&, int, int, int, formula_sdf, formula_color, double, std::vector<double>&, std::vector<typename vector3::size_type>&)’:
marching_cubes/src/marchingcubes.h:519:11: warning: typedef ‘using coord_type = typename vector3::value_type’ locally defined but not used [-Wunused-local-typedefs]
  519 |     using coord_type = typename vector3::value_type;
      |           ^~~~~~~~~~
marching_cubes/src/marchingcubes.h:520:11: warning: typedef ‘using size_type = typename vector3::size_type’ locally defined but not used [-Wunused-local-typedefs]
  520 |     using size_type = typename vector3::size_type;
      |           ^~~~~~~~~
marching_cubes/src/marchingcubes.h: In function ‘void mc::marching_cubes_super_sampling(const vector3&, const vector3&, int, int, int, int, int, int, formula, formulaX, formulaY, formulaZ, double, std::vector<double>&, std::vector<typename vector3::size_type>&)’:
marching_cubes/src/marchingcubes.h:800:11: warning: typedef ‘using coord_type = typename vector3::value_type’ locally defined but not used [-Wunused-local-typedefs]
  800 |     using coord_type = typename vector3::value_type;
      |           ^~~~~~~~~~
marching_cubes/src/marchingcubes.h:801:11: warning: typedef ‘using size_type = typename vector3::size_type’ locally defined but not used [-Wunused-local-typedefs]
  801 |     using size_type = typename vector3::size_type;
      |           ^~~~~~~~~
x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -g -fwrapv -O2 -Wl,-Bsymbolic-functions -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.9/marching_cubes/src/_mcubes.o build/temp.linux-x86_64-3.9/marching_cubes/src/marchingcubes.o build/temp.linux-x86_64-3.9/marching_cubes/src/pywrapper.o -o build/lib.linux-x86_64-3.9/marching_cubes/_mcubes.cpython-39-x86_64-linux-gnu.so
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/marching_cubes
copying build/lib.linux-x86_64-3.9/marching_cubes/smoothing.py -> build/bdist.linux-x86_64/egg/marching_cubes
copying build/lib.linux-x86_64-3.9/marching_cubes/exporter.py -> build/bdist.linux-x86_64/egg/marching_cubes
copying build/lib.linux-x86_64-3.9/marching_cubes/__init__.py -> build/bdist.linux-x86_64/egg/marching_cubes
copying build/lib.linux-x86_64-3.9/marching_cubes/_mcubes.cpython-39-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg/marching_cubes
copying build/lib.linux-x86_64-3.9/marching_cubes/numpy_smoothing.py -> build/bdist.linux-x86_64/egg/marching_cubes
byte-compiling build/bdist.linux-x86_64/egg/marching_cubes/smoothing.py to smoothing.cpython-39.pyc
byte-compiling build/bdist.linux-x86_64/egg/marching_cubes/exporter.py to exporter.cpython-39.pyc
byte-compiling build/bdist.linux-x86_64/egg/marching_cubes/__init__.py to __init__.cpython-39.pyc
byte-compiling build/bdist.linux-x86_64/egg/marching_cubes/numpy_smoothing.py to numpy_smoothing.cpython-39.pyc
creating stub loader for marching_cubes/_mcubes.cpython-39-x86_64-linux-gnu.so
byte-compiling build/bdist.linux-x86_64/egg/marching_cubes/_mcubes.py to _mcubes.cpython-39.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying PyMarchingCubes.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying PyMarchingCubes.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying PyMarchingCubes.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying PyMarchingCubes.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
marching_cubes.__pycache__._mcubes.cpython-39: module references __file__
creating dist
creating 'dist/PyMarchingCubes-0.0.2-py3.9-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing PyMarchingCubes-0.0.2-py3.9-linux-x86_64.egg
creating /usr/local/lib/python3.9/dist-packages/PyMarchingCubes-0.0.2-py3.9-linux-x86_64.egg
Extracting PyMarchingCubes-0.0.2-py3.9-linux-x86_64.egg to /usr/local/lib/python3.9/dist-packages
Adding PyMarchingCubes 0.0.2 to easy-install.pth file

Installed /usr/local/lib/python3.9/dist-packages/PyMarchingCubes-0.0.2-py3.9-linux-x86_64.egg
Processing dependencies for PyMarchingCubes==0.0.2
Finished processing dependencies for PyMarchingCubes==0.0.2

Run training and test in colab¶

In [12]:
#@title ###Define necessary functions 

#find the path of marching_cubes
import site
import glob
import sys

def find_pm_path(pm_path = '/usr/local/lib/python3.7/dist-packages/'):
  pm_path = pm_path + "/PyMarchingCubes*/"
  path = sorted(glob.glob(pm_path), reverse=True)
  if path:
      return path[0]

pm_path = find_pm_path(site.getsitepackages()[0])
# marching_cubes_path = r"/usr/local/lib/python3.7/dist-packages/PyMarchingCubes-0.0.2-py3.7-linux-x86_64.egg" #it depend on your OS but just paste the path where is mcubes
if not pm_path in sys.path:
    sys.path.append(pm_path)    
import marching_cubes as mcubes


def get_latest_dir(path):
    dir_list = os.listdir(path)
    dir_list.sort(key=lambda x:os.path.getmtime((path+"/"+x)))
    assert len(dir_list) >= 1, "there is not previous project in present output_dir, please check the path" 
    folder_name = os.path.join(output_dir, dir_list[-1])
    folder_name  = os.path.basename(folder_name)
    return folder_name

def get_latest_file(path):
    dir_list = os.listdir(path)
    dir_list.sort(key=lambda x:os.path.getmtime((path+"/"+x)))
    return dir_list[-1]

Training:¶

Parameters introduction：

  • Use Vit-B/16 in clip_model and 100 in Epoch_num for draft training; Vit-L/14 and 200 for fine training; the Vit-L/14 336 has not been tested yet.
  • check here about the performance for different clip_model , normally the model with better performance will cause more training time.
  • clip_w_h and clip_aug_copy will significantly affect the GPU RAM occupation and training speed, try to turn-in that with different combination, such as 232x12 \ 225x16 \ 168x20 \ 128x24 in 16G RAM GPU.
  • use_clip_direction_text will determine if will add the prefix to the text prompt about camera direction, such as "the font view of....". This may reduce mismath but cause over fitting.
  • Strongly recommend to select Prompt_image_direction when used Prompt_image to avoid overfitting, to assign the view direction of image prompt (even with that the training with image prompt still quite not stable).
  • set dt_gamma >= 0 to adaptive ray marching, which will accelerate the training but possibly lead to worse result.
  • set seed to -1 to get random seed
  • clip_aug is an experimental function about augment the image before input into CLIP; seems enable it to allow more details output and when turn off will train more stable.
  • random_fovy_training will apply random view angle in traning, to allow CLIP to observe the render in various scale.
  • larger camerfa_fovy amount will lead to smaller object seen in overall view.
  • set resume_project to "latest" to autoload the latest project in output_dir, otherwise indicate the specific project name.
  • The training outputs will be put in the subfolder with project_name plus timestamp when training start under output_dir;
  • Attention: Sometimes the colab UI will be stuck, but if the files under checkpoints folder is keep updating, the training is still running. You could wait to it finish then restart the kernel then run the test stage.
  • For the full usages, please check the main_nerf.py for detail.
In [17]:
# from typing_extensions import Text
#@markdown ####**Training Settings:**
Prompt_text = "KAWS as a Grim Reaper in 8k, High Detail, Skull as a Face" #@param{type: 'string'}
Prompt_image = ""  #@param{type: 'string'}
Prompt_image_direction = 'None' #@param ['None','front', 'left side', 'back', 'right side', 'top', 'bottom']
Prompt_image_direction = "" if Prompt_image_direction == 'None' else "--image_direction " + "'" + Prompt_image_direction  + "'"
Epoch_num =  500 #@param {type: 'integer'}
learning_rate = 0.0005 #@param {type: 'number'}
dt_gamma = 0 #@param {type: 'number'}
render_W_H = 384  #@param {type: 'integer'}
clip_w_h = 224  #@param {type: 'integer'}
clip_aug_copy = 8 #@param {type: 'integer'}
clip_aug = True #@param {type:"boolean"}
clip_aug = "--clip_aug " if clip_aug else ""
use_clip_direction_text = True #@param {type:"boolean"}
use_clip_direction_text = "--dir_text " if use_clip_direction_text else ""
clip_model =  'ViT-L/14' #@param ['ViT-B/32','ViT-B/16','ViT-L/14','ViT-L/14@336px','RN50x16','RN50x64']
seed =  -1 #@param {type: 'integer'}
camera_fovy = 43 #@param {type: 'integer'}
random_fovy_training = True #@param {type:"boolean"}
random_fovy_training = "--rnd_fovy" if random_fovy_training else ""
##@markdown *  set seed to -1 to get random seed
resume_train = True #@param {type:"boolean"}
resume_project = "latest"  #@param{type: 'string'}
ckpt = "latest"  #@param{type: 'string'}

#@markdown ---

#@markdown ####**Output Settings:**
output_dir = "/content/drive/MyDrive/Dreamfields3D_output/" #@param{type: 'string'}
project_name = "grim"  #@param{type: 'string'}
display_interval = 10 #@param {type: "integer"}
interval_samples = 12 #@param {type: "integer"}
save_interval_img = False #@param {type:"boolean"}
save_interval_img = "--save_interval_img" if save_interval_img else ""
save_depth = False #@param {type:"boolean"}
save_depth = "--save_depth" if save_depth else ""
ckpt_save_interval = 50 #@param {type: "integer"}
mesh_resolution = 256 #@param {type: "integer"}
mesh_threshold = 10 #@param {type: "integer"}
# assign_subdir = True #@param {type:"boolean"}

#paramater conversion
import time
import os

#convert the prompt text
Prompt_text = "'" + Prompt_text + "'"
iteration = Epoch_num * 100
#get the workspace folder

try:
    DC_dir
except NameError:
    DC_dir = '/content/dreamfields-3D/'
assert os.path.isfile (DC_dir + '/main_nerf.py'), \
"wrong working folder for dreamfileds3D, please enter the right folder and run again"

if not os.path.exists(output_dir):
    os.mkdir(output_dir)

#check the parameters
assert clip_w_h % 8 ==0, f"the value of clip_w_h should be divisible by 8, but current value {clip_w_h} is not, try to use 224 or 128"

#convert the augments
if not resume_train:
  workspace = project_name + "_" + time.strftime('%Y%m%d-%H%M%S')[4:-2]

if resume_train:
  if resume_project == "latest":
    workspace = get_latest_dir(output_dir)
  else:
    workspace = resume_project

if Prompt_image != "":
  Prompt_image = "--image " +  "'" + Prompt_image + "'"

if not os.path.exists(output_dir):
    os.makedirs(output_dir)
In [ ]:
#@title Training
%cd {DC_dir}

%run main_nerf.py --text {Prompt_text} {Prompt_image} {Prompt_image_direction} --cuda_ray --fp16 --iters {iteration} --seed {seed} \
--output_dir {output_dir} --workspace {workspace} --colab --val_int {display_interval} --val_samples {interval_samples} --w {clip_w_h} --h {clip_w_h} \
--H {render_W_H} --W {render_W_H} --fovy {camera_fovy} --clip_model {clip_model} --aug_copy {clip_aug_copy} --lr {learning_rate} --dt_gamma {dt_gamma} \
--ckpt_save_interval {ckpt_save_interval} {save_interval_img} {use_clip_direction_text} {clip_aug} {random_fovy_training} --ckpt {ckpt} {save_depth} 
%cd {DC_dir}

++> Evaluate at epoch 300 ...

loss=-0.8714 (-0.8497): :   8% 1/12 [00:00<00:01,  6.50it/s]
[density grid] min=0.0000, max=1.6407, mean=0.0169, occ_rate=0.058 | [step counter] mean=507607

loss=-0.8484 (-0.8826): :  92% 11/12 [00:00<00:00, 13.08it/s]

0

1

2

3

4

5

6

7

8

9

10

11

loss=-0.8484 (-0.8826): : 100% 12/12 [00:01<00:00,  9.66it/s]

++> Evaluate epoch 300 Finished.

[INFO] New best result: -0.8745408107837042 --> -0.8825522661209106

==> Start Training Epoch 301, lr=0.000493 ...

[density grid] min=0.0000, max=1.7186, mean=0.0172, occ_rate=0.058 | [step counter] mean=507607

loss=-0.8084 (-0.8665): : 100% 100/100 [00:11<00:00,  8.82it/s]

==> Finished Epoch 301.

==> Start Training Epoch 302, lr=0.000493 ...

[density grid] min=0.0000, max=1.7947, mean=0.0176, occ_rate=0.058 | [step counter] mean=448596

loss=-0.7118 (-0.8454): : 100% 100/100 [00:11<00:00,  8.78it/s]

==> Finished Epoch 302.

==> Start Training Epoch 303, lr=0.000493 ...

[density grid] min=0.0000, max=1.8556, mean=0.0188, occ_rate=0.058 | [step counter] mean=281461

loss=-0.8119 (-0.8468): :  57% 57/100 [00:06<00:04,  8.69it/s]

Test:¶

Generate image / video / mesh with latest pretrained checkpoint file.

  • set test_project to "latest" to autoload the latest project in output_dir, otherwise indicate the folder name of specific project (include the auto generate timestamp).
  • test_samples will control how many images will be generate, but only 20 will be displayed in colab, please check the output folder to view all images.
  • Check the save_video to generate the 360° video. The Video will be 20 FPS and the total frames of video will be as same as test_samples.
  • Check the save_mesh to generate obj and ply 3D model by marching cube algorithm.
  • The output models contain vertex colour, and could be directly view in meshlab and Rhino3D. For viewing the colour in Blender, please import the ply model (OBJ also work on latest blender 3.3), then create a new material for it, and plug a Color Attribute node into the new material in shader editor, then you should see the vertex colour.
In [ ]:
# from typing_extensions import Text
#@markdown ####**Test Settings:**
test_project = "latest"  #@param{type: 'string'}
ckpt = "latest"  #@param{type: 'string'}
test_samples = 240 #@param {type: "integer"}
render_W_H = 1024  #@param {type: 'integer'}
mesh_resolution = 256 #@param {type: "integer"}
mesh_threshold = 10 #@param {type: "integer"}
save_depth = False #@param {type:"boolean"}
save_depth = "--save_depth" if save_depth else ""
save_video = False #@param {type:"boolean"}
save_video = "--save_video" if save_video else ""
save_mesh = True #@param {type:"boolean"}
save_mesh = "--save_mesh" if save_mesh else ""

#convert the parameters
if test_project == "latest" :
  workspace = get_latest_dir(output_dir)
else:
  workspace = test_project
In [ ]:
#@title Testing
#@ markdown Will generate images and mesh output use the latest checkpoint, even when the training has not finished.
%cd {DC_dir}
%run main_nerf.py --text {Prompt_text} --cuda_ray --fp16 --iters {iteration} --seed {seed} \
--output_dir {output_dir} --workspace {workspace} --colab --val_int {display_interval} --w {clip_w_h} --h {clip_w_h} \
--H {render_W_H} --W {render_W_H} --fovy {camera_fovy} --clip_model {clip_model} --aug_copy {clip_aug_copy} --dt_gamma 0 \
--ckpt_save_interval {ckpt_save_interval} {save_interval_img} {use_clip_direction_text} {clip_aug} --ckpt {ckpt} {save_depth} \
--test_samples {test_samples} --mesh_res {mesh_resolution} --mesh_trh {mesh_threshold}  {save_video} {save_mesh} \
--test
%cd {DC_dir}
In [ ]:
#@title display latest RGB video
from IPython.display import HTML
from base64 import b64encode
path2video =  f"{output_dir}/{workspace}/videos/" + get_latest_file (f"{output_dir}/{workspace}/videos/")
def show_video(video_path, video_width = 600):
   
  video_file = open(video_path, "r+b").read()
 
  video_url = f"data:video/mp4;base64,{b64encode(video_file).decode()}"
  return HTML(f"""<video width={video_width} controls><source src="{video_url}"></video>""")
 
show_video(path2video)
Out[ ]: